DATA ENGINEERING + AI / END-TO-END PLATFORM
Polaris collects Data & AI job postings from three Vietnamese recruitment sources, transforms raw web pages into a traceable PostgreSQL model, and ranks CV-to-job fit with vector embeddings — served through a Next.js portal with market dashboards and scheduled alerts.
01 / CONTEXT & PROBLEM
Vietnam's data-job market is scattered across boards with incompatible structures, duplicate postings, and no way to ask the question that matters: "which of these jobs actually fits my CV?" Keyword matching rewards buzzword overlap, not real fit.
The goal: one traceable data model where every normalized record can be traced back to its preserved raw HTML, plus semantic matching that ranks jobs by meaning — wrapped in a portal a job-seeker would actually use.
02 / ARCHITECTURE
3 job boards
daily crawls via GitHub Actions
→
MinIO raw layer
original HTML preserved · full traceability
→
BeautifulSoup parsers
source-specific fields · transactional upserts
→
PostgreSQL + dbt + pgvector
staging & analytical models · 384-dim MiniLM embeddings · cosine similarity
→
Next.js portal
AI matching · dashboards · hourly alerts
Dagster orchestrates · Redis distributed locks prevent duplicate runs · 9-container Docker Compose stack incl. Cube, Cube Store, Metabase
03 / KEY DECISIONS
Preserve raw HTML before parsing anything
Job boards change their markup constantly. Keeping the original page in MinIO means a parser bug is a re-run, not data loss — and every normalized field stays auditable back to its source.
pgvector inside PostgreSQL, not a separate vector DB
Embeddings live next to the relational model, so semantic ranking joins directly with structured filters (location, level, salary) in one SQL query — no sync problem between two stores.
Redis distributed locks around scheduled jobs
Daily crawls and hourly alert deliveries are guarded by locks and authenticated internal APIs — overlapping runs can't double-send emails or double-insert postings.
Quality gates across both languages
21 automated tests cover TypeScript and Python: linting, formatting, dbt parsing, Prisma validation, and Dagster definitions — the polyglot stack fails fast in CI, not in production.
04 / RESULTS
A recorded dataset of 234 postings visualized by location, experience level, hiring trend, and company demand — with daily crawling and hourly SMTP-tracked alert delivery running unattended.
05 / WHAT I'D DO DIFFERENTLY
Semantic ranking works, but "works" needs a benchmark: a labeled set of CV-job pairs to measure precision against keyword baselines, and A/B evidence that embedding similarity predicts real fit. That evaluation harness comes before any model upgrade.
I'd also grow the dataset — 234 recorded postings proves the pipeline; market-level claims need months of accumulation, which the daily automation is designed to deliver.